Add `xxhash64` support for nested types #2575

ustcfy · 2024-11-06T06:32:12Z

Contribute to #2574

This PR adds support for xxhash64 with struct , list and map types.

The hardcoded expected values in the tests are generated using Spark.
For example, you can produce the same results as in testXXHash64ListOfStruct using the following code.

import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row

val schema = StructType(Seq(
    StructField("listofstruct",
        ArrayType(
            StructType(Seq(
                StructField("string", StringType, true),
                StructField("int", IntegerType, true),
                StructField("double", DoubleType, true),
                StructField("float", FloatType, true),
                StructField("bool", BooleanType, true)
            )),
            true
        )
    )
))

val data = Seq(
    Row(List()),
    Row(List(Row("a", 0, 0.0, 0f, true))),
    Row(List(Row("B\n", 100, 100.0, 100f, false), Row("dE\"\u0100\t\u0101 \ud720\ud721", -100, -100.0, -100f, null))),
    Row(List(Row("A very long (greater than 128 bytes/char string) to test a multi hash-step data point " +
        "in the MD5 hash function. This string needed to be longer.", Int.MinValue, java.lang.Double.longBitsToDouble(0x7ff0000000000001L), java.lang.Float.intBitsToFloat(0xff800001), false))),
    Row(List(Row(null, Int.MaxValue, java.lang.Double.longBitsToDouble(0x7fffffffffffffffL), java.lang.Float.intBitsToFloat(0xffffffff), true), Row(null, null, null, null, null))),
    Row(null)
)

val df = spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
df.columns.foreach(c => println(s"$c => ${df.select(xxhash64(col(c))).collect.mkString(",")}"))

You will get the following output:

listofstruct => [42],[7451748878409563026],[948372773124634350],[8444697026100086329],[-5888679192448042852],[42]

src/main/cpp/src/xxhash64.cu

src/test/java/com/nvidia/spark/rapids/jni/HashTest.java

src/main/cpp/src/xxhash64.cu

res-life · 2024-11-06T07:48:39Z

Please add more cases, LGTM for the function.

res-life · 2024-11-06T08:22:53Z

LGTM

res-life · 2024-11-06T08:24:35Z

build

src/main/cpp/src/xxhash64.cu

revans2

looks good, but I am no expert on the CUDA side of things.

revans2 · 2024-11-06T20:11:20Z

src/main/cpp/src/xxhash64.cu

+                                            cudf::size_type row_index) const noexcept
+      {
+        if (_check_nulls && col.is_null(row_index)) { return _seed; }
+        return XXHash_64<T>{_seed}(col.element<T>(row_index));


Can we have a follow on issue to make the hash function a template? We have the same issues with all hash functions we want to support, so having generic code would be nice.

firestarman

Look good to me, but a nit.

src/main/cpp/src/xxhash64.cu

ttnghia

Sorry, this PR has similar implementation with the HiveHash one, so both need to address the issue with shallow stack size unless we want to target this for 24.12.

The base branch was changed.

Signed-off-by: ustcfy <[email protected]>

Co-authored-by: Chong Gao <[email protected]>

Signed-off-by: ustcfy <[email protected]>

Co-authored-by: Chong Gao <[email protected]>

Signed-off-by: ustcfy <[email protected]>

Signed-off-by: Yan Feng <[email protected]>

ustcfy requested review from firestarman and res-life November 6, 2024 06:32

ustcfy self-assigned this Nov 6, 2024

ustcfy changed the title ~~Add xxhash64 support for nested types~~ Add xxhash64 support for nested types Nov 6, 2024

ustcfy requested a review from nvdbaranec November 6, 2024 06:52